record(GEMMA4-ROCM-KEEP): the 2x R9700 KEEP recipe, and the denominator it still lacks - #676

Thanks for this — and welcome. Reviewed as part of a sweep over the open external PRs.

Before the findings, the part that matters: all eight checkers you list as passing locally do pass — I ran every one at your head SHA and confirmed it. Everything below is something no checker in this repo looks for, so none of it is a diligence failure on your part. Several of your calls are ones this project learned the hard way: disclosing the dropped 1170 first-rep with its cause rather than quietly taking the median, per-depth rep counts, explicitly writing "does not claim that bar", refusing to name an LLVM defect from a spill count, and not touching STATUS.md when no lifecycle moved. The RADV/ACO matched-instrument comparison in your second commit is the most reusable thing in the PR.

The blocker is that the recipe cannot be reproduced from main.

Four of the decode knobs are read by no production code in this tree:

git log -S'VT_ATTN_DECODE_KV_SPLITS' and git log -S'SLIDE_WARPS' both put the production consumers on dd1288095, whose subject line is "wip(lab): decode/BPE/structural-tags stack (not for mudler merge)".

Meanwhile every prefill item — SharedK-WMMA on, FLASH off, GEMM_M=2048, PEER_ACT=1, batch MoE T>=64 — is already the shipped default on main. So read against this tree the recipe is the default configuration plus four inert strings, and docs/USAGE.md:2125 now tells readers of a public page to set two variables that do nothing.

I don't think the numbers are wrong. The likely story is honest measurements of your working branch written up against the wrong baseline. But a record whose whole value is reproducibility has to name the tree it describes, and the spec header still says PR tip feat/gemma4-rocm-fp8-split with no SHA. Either state the exact revision and mark the decode knobs lab-branch-only, or drop them and re-state the recipe as "the defaults" — which on main it is.

Why this was easy to miss, and not your fault: test_gemma4_rocm_fp8_seams.cpp:69 asserts EnvInt("VT_ATTN_DECODE_KV_SPLITS", 16) == 16 with the variable unset. That is default == default — it tests the helper's fallback, not the product, and would pass identically if the env var had never existed. Its own comment half-concedes it. That is a pre-existing defect of ours and I'm filing it separately; it is why three non-existent knobs look real from inside the repo.

Second blocker: no issue. AGENTS.md wants one linked in three places that agree — the roadmap issue table, the row's spec, and the PR body. The spec header cites (#317), which is a merged PR rather than an issue; the roadmap has no entry; the body has none. Also grep -rn "GEMMA4-ROCM-KEEP" matches nothing in the tree, so the row ID your branch name asserts doesn't exist yet.

On the denominator. The comparison arm is "Vulkan Q8" with the stack unnamed. The prior entry for this same box labelled it llama.cpp Vulkan Q8 … separate stack (.agents/benchmark-record.md:16185), and docs/ROCM.md:300-303 says of this exact milestone: "The bar is vLLM, not llama.cpp." It's also not quant-matched — ours FP8, the bar Q8_0 — while derived ratios (~1.74x / 2.47x) are published across that boundary.

Your "does not claim that bar" framing is the right instinct and it is one sentence from being fully admissible. There's a template already in the tree at docs/BENCHMARKS.md:37: "no vLLM denominator (pin lacks muse_glimmer); SECONDARY llama.cpp, same GGUF, idle GB10 | vLLM axis is an OPEN GAP." Name the stack, label it secondary, declare the vLLM axis open. Note docs/ROCM.md:216-231 documents two working vLLM-ROCm oracle recipes on RDNA4, so if it turns out vLLM genuinely can't serve this checkpoint on 2x R9700, that's worth saying explicitly rather than leaving unstated.

Placement. The measurement landed on USAGE.md, ROCM.md, FEATURES.md and the spec, but not on either surface this project keeps measurements on. .agents/benchmark-record.md is append-only — no merge-lock excuse — and its header says "read it before re-running a lever." Your five rejected levers are exactly what it exists for, and they aren't in it. docs/BENCHMARKS.md:452 still reads PENDING: no binding throughput number with no reconciliation.

docs/FEATURES.md:335 is the sharpest one: the edit removes Env VT_GEMMA4_*/VT_ATTN_*, seam test_gemma4_rocm_fp8_seams — the two facts that belong on a feature surface — to make room for a bare absolute with no denominator and none of your own caveat. The only other throughput in that table carries its vLLM ratio.

Two smaller things worth having:

• docs/ENVIRONMENT.md:198 documents GEMM_M default 256, range 16..2048. The code says default 2048, range 16..8192 (gemma4_moe.cpp:1016-1021). Your new sentence reads as a lab override of a documented 256 when 2048 is what the binary already does. Worth correcting in the same edit — check-env-doc.py verifies documentation exists, not that it's accurate.
• The ~18k and ~42k rows don't state VT_GEMMA4_GPU0_HEADROOM_GB, and docs/ENVIRONMENT.md:194 records for this exact board that 8 survives 16k+ prefill while 6 OOMs at ~11k. Either the default was used, or the recipe is incomplete for its two deepest rows.

Please keep the residual section as-is. The HIP cm1 339-spill vs ACO 0-spill contrast on the same silicon, with the explicit refusal to call it a named defect, is disciplined work and it should not be lost when the spec is compacted — I'd suggest an issue so it survives independently.

Decode regime would help too: 55.5 / 49.1 t/s with no concurrency, depth or batch size can't be compared to anything, including a future vLLM leg. Your prefill depth curve is exactly right; decode just needs the same treatment.

Nothing here needs new measurements. Happy to look again once the recipe names its tree.

Following up on the review above — there is one question only you can answer, and it blocks the rest.

The question: which tree produced these numbers?

The KEEP recipe lists four decode knobs, and none of them is read by production code on main:

git log -S puts their production consumers on dd1288095, whose own subject line is "wip(lab): decode/BPE/structural-tags stack (not for mudler merge)".

Meanwhile every prefill item — SharedK-WMMA on, FLASH off, GEMM_M=2048, PEER_ACT=1, batch MoE T>=64 — is already the shipped default on main.

I am not suggesting the measurements are wrong. The most likely explanation is that they are honest numbers from your working branch, written up against main as the baseline. But a record whose whole value is reproducibility has to name the tree it describes, and the spec header still says PR tip feat/gemma4-rocm-fp8-split with no SHA.

So: which revision were the 2112 / 2014 / 1705 / 1099 prefill figures and the 55.5 / 49.1 decode figures taken on? Once that is stated, there are two clean ways forward and either is fine:

• State the SHA and mark the decode knobs lab-branch-only, so a reader knows the recipe is not reproducible from main yet; or
• Drop the four decode knobs and re-state the recipe as "the shipped defaults", which on main is what it is.

The practical urgency is that docs/USAGE.md:2125 currently tells readers of a public page to set two variables that do nothing on main.

Not your fault, and worth saying

tests/vt/test_gemma4_rocm_fp8_seams.cpp:69 asserts EnvInt("VT_ATTN_DECODE_KV_SPLITS", 16) == 16 with the variable unset — that is default == default. It tests the helper's fallback, not the product, and would pass identically if the env var had never existed. That is a defect of ours and it is why three non-existent knobs look real from inside the repo. I am filing it separately.

The smaller items, unchanged from the earlier review

1. No issue, in any of the three required places. The spec header cites (#317), which is a merged PR rather than an issue; the roadmap has no entry; the PR body has none. grep -rn "GEMMA4-ROCM-KEEP" also matches nothing, so the row ID your branch name asserts does not exist yet.
2. The numbers are on the wrong surfaces. docs/BENCHMARKS.md and the append-only .agents/benchmark-record.md are both untouched. The latter exists precisely so nobody re-runs a closed lever — your five rejected levers are exactly what it is for. benchmark-record.md is append-only, so there is no merge-lock reason to avoid it.
3. docs/FEATURES.md:335 trades information for a number: the edit deletes Env VT_GEMMA4_*/VT_ATTN_*, seam test_gemma4_rocm_fp8_seams — the two facts that belong on a feature surface — for a bare absolute with no denominator.
4. Name the comparison stack and label it. "Vulkan Q8" does not say whose Vulkan. The prior entry for this same box labelled it llama.cpp Vulkan Q8 … separate stack, and docs/ROCM.md:300-303 says of this milestone: "The bar is vLLM, not llama.cpp." Your "does not claim that bar" framing is the right instinct and is one sentence from being fully admissible — there is a template at docs/BENCHMARKS.md:37 (the Muse Glimmer row) that names the absence, labels the secondary, and declares the vLLM axis an OPEN GAP.
5. docs/ENVIRONMENT.md:198 documents GEMM_M default 256, range 16..2048; the code says default 2048, range 16..8192 (gemma4_moe.cpp:1016-1021). Worth correcting in the same edit — your sentence reads as a lab override of a documented 256 when 2048 is already what the binary does.
6. Decode regime: 55.5 / 49.1 t/s with no concurrency, depth or batch size cannot be compared to anything. Your prefill depth curve is exactly right; decode just needs the same treatment.

Please keep the residual section as-is

The HIP cm1 339-spill vs ACO 0-spill contrast on the same silicon, with the explicit refusal to call it a named LLVM defect, is the most reusable thing in this PR. I would suggest an issue for it so it survives the spec being compacted later.

Nothing here needs new measurements — just the provenance question answered.

This is a legitimate record-only PR — the record is the work here, not a
restatement of something that already landed — and the best thing in it deserves
to be said first.

The RADV/ACO comparison is the most valuable result in this branch. Matched
instrument, same silicon, same problem: ACO at 0 spilled VGPR and 0 scratch at
VGPR=256, against HIP-LLVM at 235–339 spills. That is a real, traceable
hypothesis for where the remaining gap lives, and you explicitly refuse to
over-claim it as a named LLVM defect. It should not be buried in a spec — it
warrants its own issue.

On "plateau": in substance you are within the rule, but the title is not.
AGENTS.md forbids declaring a ceiling. The body does the opposite of that — it
says outright "It does not close that bar", keeps the gap open with numbers
(3503 @11k / 2714 @42k against 2014 / 1099), and names the next hypothesis. That
is more disciplined than most internal rows. But the title is what survives
into git log, and "measured SharedK-WMMA plateau" alone reads as "this is where
ROCm tops out". Something like "the measured SharedK-WMMA recipe on 2x R9700,
and the ACO spill gap above it" keeps the result and drops the ceiling reading.
Same for "reliable plateau" in docs/USAGE.md, which is the page a user reads.

The blocker is not the wording, though — it is that the recipe does not run on
this tree. Checked against main at 281e6a120:

• VT_ATTN_DECODE_KV_SPLITS, VT_ATTN_DECODE_SLIDE_SPLITS and
  VT_ATTN_DECODE_SPLIT_WARPS appear nowhere in src/ or include/ — only
  in tests/vt/test_gemma4_rocm_fp8_seams.cpp:59-71.
• VT_ATTN_DECODE_SLIDE_WARPS matches nothing anywhere.
• VT_GEMMA4_PEER_ACT as spelled matches nothing; the real knob is
  VT_GEMMA4_PREFILL_PEER_ACT (gemma4_moe.cpp:1024).

So docs/USAGE.md would tell readers to export four decode knobs the binary
never reads. The spec header says PR tip feat/gemma4-rocm-fp8-split with no
SHA, and a record whose whole value is reproducibility has to name the tree it
describes. Only you know which revision these numbers came from — that is the
one thing we cannot repair from here.

Relatedly, docs/ENVIRONMENT.md:227 is edited but its two errors are left in
place: it says default 256, range 16..2048, while
gemma4_moe.cpp:1016-1021 uses default 2048 and range 16..8192. The
added "Lab KEEP on dual R9700 uses 2048" then reads as a lab override of a
documented 256 when 2048 is simply what the binary does. The same edit also
deletes a previously recorded measurement (lab 512 ~+37% prefill vs 64) —
AGENTS.md asks that evidence be moved, not dropped.

The measurement is not yet admissible as published. Against AGENTS.md
§Gates and .agents/benchmarking.md, what is missing is: the pinned oracle on
both sides (there is no vLLM-ROCm leg at all), a denominator of any kind, a
quant match (ours FP8 vs the comparator's Q8_0, with ratios published across that
boundary), the build and run recipe, revisions, model hashes, contention state,
and an idle-host same-binary A/B. docs/BENCHMARKS.md:479 still reads
PENDING: no binding throughput number and is not reconciled, and the
append-only .agents/benchmark-record.md — which exists precisely so the five
levers you rejected are not re-run by someone else — is untouched.

The axis that is missing matters more than the word "plateau". Every gap you
keep open is against our own Vulkan backend and llama.cpp. The axis the project
actually gates on, vLLM-ROCm, is simply absent — neither measured nor declared
open — while docs/ROCM.md:216-231 documents two working vLLM-ROCm oracle
recipes on RDNA4. An unstated axis is worse than a stated ceiling, because
nothing is left pointing at it. docs/BENCHMARKS.md:37 (the Muse Glimmer row)
is the template: name the absence, label the llama.cpp/Vulkan comparison
SECONDARY, and write "vLLM axis is an OPEN GAP". If vLLM genuinely cannot serve
this checkpoint on 2x R9700, say that — it is a useful finding in itself.

Two CI notes so they are not re-derived. Your sanitize-cpu (address,undefined) red was never yours: it is
ltx2_loader.cpp:1325:91: runtime error: load of misaligned address ... 'const uint16_t' in test_ltx2_video, which main itself carried until fc903b8dd
(#674) landed about 15 hours after this branch's base. A +49/-5 markdown diff
cannot reach that TU. And windows-msvc-* were red on every open PR from a
pre-existing break; that queue has since been cleared down to #584.

What would make this landable: the revision the numbers came from, the knob
list corrected to what the shipped binary reads (for every prefill item, that is
simply the defaults), the ENVIRONMENT.md row fixed rather than annotated, an
append to .agents/benchmark-record.md, docs/BENCHMARKS.md:479 reconciled with
the vLLM axis named as an open gap, and the title reworded. The measurements
themselves are worth keeping — the depth curve is exactly the right shape, and
dropping the 1170 first rep rather than quietly medianing it is good practice.

@bakon11 — I pushed the fixable parts to row/GEMMA4-ROCM-KEEP-MEASURED (c08af5271 → f649859ea, rebased onto affc2a7fd; the branch was 215 commits behind, both of your commits keep their authorship, no merge commit) and rewrote the PR body and title. One thing is left and only you can answer it.

The blocker: four knob names in the published recipe are read by nothing in this tree.

$ git grep -n 'VT_ATTN_DECODE_KV_SPLITS\|VT_ATTN_DECODE_SLIDE_SPLITS\|VT_ATTN_DECODE_SPLIT_WARPS\|VT_ATTN_DECODE_SLIDE_WARPS' -- src include
(nothing)

The first three appear only in tests/vt/test_gemma4_rocm_fp8_seams.cpp:59-71. The fourth appears nowhere at all. VT_GEMMA4_PEER_ACT also matches nothing — the product knob is VT_GEMMA4_PREFILL_PEER_ACT (gemma4_moe.cpp:1026), and I corrected that one plus PREFILL_GEMM_M → VT_GEMMA4_PREFILL_GEMM_M since those are unambiguous.

This is not you inventing names. It is #845: that seam test asserts EnvInt(name, 16) == 16 with the variable unset, which is a tautology that passes whether or not the knob exists. Three names nothing reads have looked real ever since.

So the recipe cannot be followed as written, and I cannot tell which way it fails, because the spec's **Status:** line says PR tip feat/gemma4-rocm-fp8-split with no SHA. Either:

• those four splits were live on the tree you measured and it is not this one — in which case the numbers do not describe main and we need the SHA; or
• they were already inert on 2026-08-13 and the shipped defaults produced 2014 @11k — in which case the recipe just loses four lines and the numbers stand.

Only you know. The SHA, and which of the two, is what unblocks this. I did not guess: the spec keeps your full as-run recipe with that question attached, and docs/USAGE.md publishes only the knobs a reader can actually set, saying outright that the decode figure is not reproducible from them.

What I fixed.

docs/ENVIRONMENT.md said VT_GEMMA4_PREFILL_GEMM_M defaults to 256 over 16..2048. gemma4_moe.cpp:1016-1021 accepts 16..8192 and returns 2048, and has since 2026-08-10. That row was already wrong on main; your edit touched it, so it is repaired in flow. Your edit also dropped a recorded measurement — lab 512 ~+37% prefill vs 64 — and AGENTS.md says move evidence, never drop it, so it is back, next to the 512→2048 ~+80 eng @11k result that explains why the default is what it is.

docs/FEATURES.md is back to main's text. A throughput figure's home is docs/BENCHMARKS.md under the projection table; this PR changes no feature, backend or quantization surface; and fitting the number into a cell already 219 of its 220 characters cost the row its VT_GEMMA4_*/VT_ATTN_* pointer and the test_gemma4_rocm_fp8_seams seam name.

I retitled. The body always said the right thing — "does not close that bar", with the ACO disparity named as the next hypothesis — but on a squash merge only the title reaches git log, and "plateau" reads there as exactly the ceiling claim AGENTS.md forbids. It is now "the 2x R9700 KEEP recipe, and the denominator it still lacks".

The old sanitize-cpu (address,undefined) red was test_ltx2_video, from a lane 215 commits ahead of your old base; the rebase carries it away.

On the denominator, since it is the other half of why this cannot land as a perf record.

Every number here is engine-side with nothing on the other side. Under AGENTS.md "Gates" that is not a throughput result, which is why docs/BENCHMARKS.md:506 still reads PENDING: no binding throughput number and I left it alone — it is accurate, and changing it without a denominator would be the wrong kind of tidy. I also deliberately did not append to .agents/benchmark-record.md: it is the append-only log, and a figure that enters it gets quoted afterwards as measured.

The oracle is not hypothetical — docs/ROCM.md §5 documents two working Docker vLLM-ROCm recipes on this hardware family, the second building this project's pinned 555967922 inside rocm/vllm-dev:base in about 6.5 minutes. Same model, quantization, prompt set, request shape, concurrency and cache policy, one idle box, ratio per depth. That is the run that turns this from a recipe into a benchmark, and it is a genuinely good result waiting for its other half.

Both items are under ## Owed in .agents/specs/gemma4-rocm-fp8-moe.md, along with a note that the Vulkan Q8 gap (1.74x @11k, 2.47x @42k) stays open with your HIP-LLVM-vs-ACO spill finding as the next traceable hypothesis — 339 spills against ACO's 0 is a real lead, and it should not be filed as a ceiling.

Answering the blocker (coordinator for bakon11's lab, with the lab records in hand):

It is your option 1 — the four decode knobs were live on the measured tree, which is not this one.

• Measured tree: lab checkout at tip 9a009eb6 (feat/gemma4-fp8-prefill-reliability) plus local uncommitted lab changes — the spec's missing SHA is 9a009eb6, with the honest caveat that the as-run binary (built 2026-08-13) included lab-only dirt beyond it.
• All four VT_ATTN_DECODE_* knobs are read there (rocm_paged_attn.hip ~4171–4278: SGLang-style multi-CTA KV-split decode) — that decode work has not been upstreamed, so the decode figures (55.5 t/s @ temp0 / 49.1 @ 0.7) describe the lab tree, not main. Dropping the four lines from the main-facing recipe is correct, and the decode figure should stay marked non-reproducible-from-main for now.
• One more thing your reproduction attempt will hit: the prefill figures also do not currently reproduce on main — because of ROCm: PagedAttnPrefillSharedKWmma has never launched — the host-side #if defined(VT_ROCWMMA_OK) guard is always false, so d=256/d=512 prefill silently runs the scalar kernel #785 (the host-side VT_ROCWMMA_OK guard compiles the SharedK-WMMA prefill launches out; main silently runs scalar prefill at d256/d512). The lab tree hard-enables the path, which is where 2112/2014/1705/1099 come from. We've taken ROCm: PagedAttnPrefillSharedKWmma has never launched — the host-side #if defined(VT_ROCWMMA_OK) guard is always false, so d=256/d=512 prefill silently runs the scalar kernel #785 and are preparing the runtime-arch-dispatch fix; once it lands, the prefill numbers should be reachable from main with the published knobs.

So: numbers stand as lab measurements with SHA 9a009eb6+dirty; recipe on main loses the four decode lines; prefill reproducibility on main is gated on #785. Thanks for the careful teardown — the #845 tautology catch explains exactly how inert names survived.